Information processing apparatus, information processing method, and endoscope system for processing images based on surgical scenes

ABSTRACT

An information processing apparatus, an information processing method, and an endoscope system capable of providing an optimal video image to an operator in accordance with surgical scenes are provided. A processing mode determination unit determines, in accordance with surgical scenes, a processing mode for an in-vivo image captured by an imaging apparatus including an imaging element arranged so as to enable pixel shift processing, and an image combining unit processes an image output from the imaging apparatus, in accordance with the processing mode. The present technology is applicable to, for example, an endoscope system for imaging a living body with an endoscope.

TECHNICAL FIELD

The present technology relates to an information processing apparatus,an information processing method, and an endoscope system, and moreparticularly relates to an information processing apparatus, aninformation processing method, and an endoscope system capable ofproviding optimal video images to an operator in accordance withsurgical scenes.

BACKGROUND ART

As a technique for obtaining a high definition image, a techniquereferred to as pixel shift processing is known (for example, refer toPatent Documents 1 and 2).

CITATION LIST Patent Document

Patent Document 1: Japanese Patent Application Laid-Open No. 2010-268244

Patent Document 2: Japanese Patent Application Laid-Open No. 2011-95073

SUMMARY OF THE INVENTION Problems to be Solved by the Invention

In a case where an imaging system capable of performing pixel shiftprocessing is used as an endoscopic imaging system in an endoscopicsurgery system that supports an operator or the like who performs asurgery using an endoscope, there are cases where providing highdefinition video images is not appropriate depending on surgical scenes,and this produces a demand for a technique capable of providing optimalvideo images to the operator in accordance with surgical scenes.

The present technology has been made in view of this situation and isintended to be able to provide an optimal video image to an operator inaccordance with surgical scenes.

Solutions to Problems

An information processing apparatus according to the present technologyincludes: a processing mode determination unit that determines, inaccordance with surgical scenes, a processing mode for an in-vivo imagecaptured by an imaging apparatus including an imaging element arrangedso as to enable pixel shift processing; and a processing unit thatprocesses an image output from the imaging apparatus, in accordance withthe processing mode.

An information processing method according to the present technology isan information processing method of an information processing apparatus,the method including steps of: determining, by the informationprocessing apparatus in accordance with surgical scenes, a processingmode for an in-vivo image captured by an imaging apparatus including animaging element arranged so as to enable pixel shift processing; andprocessing an image output from the imaging apparatus, by theinformation processing apparatus in accordance with the processing mode.

An endoscope system according to the present technology is an endoscopesystem including an endoscope and an information processing apparatus,in which the endoscope includes: an imaging element arranged so as toenable pixel shift processing; and a control unit that controls theimaging element, the information processing apparatus includes: aprocessing mode determination unit that determines a processing mode foran in-vivo image captured by the endoscope, in accordance with surgicalscenes; and a processing unit that processes an image output from theendoscope, in accordance with the processing mode, and the control unitcontrols the imaging element in accordance with the processing mode.

In the information processing apparatus, the information processingmethod, and the endoscope system according to the present technology, aprocessing mode for an in-vivo image captured by an imaging apparatusincluding an imaging element arranged so as to enable pixel shiftprocessing is determined in accordance with surgical scenes, and animage output from the imaging apparatus is processed in accordance withthe processing mode.

Effects of the Invention

According to the present technology, it is possible to provide anoptimal video image to an operator in accordance with surgical scenes.

Note that effects described herein are non-restricting. The effects maybe any effects described in the present disclosure.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram illustrating an embodiment of an endoscopic surgerysystem according to the present technology.

FIG. 2 is a diagram illustrating a detailed exemplary configuration of aCCU and an endoscope according to a first embodiment.

FIG. 3 is a view illustrating an outline of a single-chip pixel shiftmethod.

FIG. 4 is a diagram illustrating an exemplary correspondence table in acase where a single-chip sensor is applied.

FIG. 5 is a diagram illustrating an outline of a three-chip pixel shiftmethod.

FIG. 6 is a diagram illustrating an exemplary correspondence table in acase where a three-chip sensor is applied.

FIG. 7 is a timing chart illustrating RGB signal output timings for eachof processing modes.

FIG. 8 is a diagram illustrating an exemplary configuration of an imagecombining unit corresponding to an HFR mode.

FIG. 9 is a flowchart illustrating an image combining processing flowaccording to the first embodiment.

FIG. 10 is a diagram illustrating a detailed exemplary configuration ofa CCU and an endoscope according to a second embodiment.

FIG. 11 is a flowchart illustrating an image combining processing flowaccording to the second embodiment.

FIG. 12 is a diagram illustrating an exemplary configuration of acomputer.

MODE FOR CARRYING OUT THE INVENTION

Hereinafter, embodiments of the present technology will be describedwith reference to the drawings. Note that the description will be givenin the following order.

1. System configuration

2. First Embodiment: determination of processing mode using externalsignal

(1) Case where single-chip sensor is applied

(2) Case where three-chip sensor is applied

3. Second embodiment: determination of processing mode by scenerecognition

4. Modification

5. Configuration of computer

1. System Configuration

(Exemplary Configuration of Endoscopic Surgery System)

FIG. 1 is a diagram illustrating an embodiment of an endoscopic surgerysystem according to the present technology.

An endoscopic surgery system 10 is a system arranged in a surgery roomand configured to support an operator performing an endoscopic surgeryon an affected site included in an abdomen 30 of a patient lying on apatient bed 20, for example.

In FIG. 1, the endoscopic surgery system 10 includes a camera controlunit (CCU) 101, a light source apparatus 102, a treatment tool apparatus103, a pneumoperitoneum apparatus 104, a display apparatus 105, arecorder 106, and a cart 11 on which a printer 107 is mounted. Moreover,the endoscopic surgery system 10 includes an endoscope (laparoscope)111, an energy treatment tool 112, and a foot switch 121. Additionally,tools such as trocars 131 to 134 and forceps 135 are used by anoperator, or the like, at the time of surgery.

In the endoscopic surgery system 10, the CCU 101 is connected to theendoscope 111 via a camera cable. Note that the CCU 101 may bewirelessly connected to the endoscope 111. The CCU 101 receives anintraoperative image captured by the endoscope 111 and transmitted viathe camera cable, and supplies the received image to the displayapparatus 105.

The display apparatus 105 includes a stationary 2D display, a head mountdisplay, and the like. The display apparatus 105 displays anintraoperative image, or the like, supplied from the CCU 101. Inaddition, the CCU 101 supplies the received intraoperative image to therecorder 106 or the printer 107, as necessary.

The light source apparatus 102 is connected to the endoscope 111 via alight guide cable. The light source apparatus 102 switches light ofvarious wavelengths and emits the light to the endoscope 111.

The treatment tool apparatus 103 is a high frequency output device, andis connected to the energy treatment tool 112 and the foot switch 121via a cable. The treatment tool apparatus 103 outputs a high frequencycurrent to the energy treatment tool 112 in response to an operationsignal supplied from the foot switch 121.

The pneumoperitoneum apparatus 104 includes insufflation and suctionunits and supplies air into the abdomen 30 from a hole of the trocar 133as an opening tool attached to an abdominal wall of the abdomen 30.

The recorder 106 records an intraoperative image supplied from the CCU101. The printer 107 prints the intraoperative image supplied from theCCU 101.

The endoscope 111 includes an imaging unit and an optical system such asan illumination lens. The endoscope 111 is inserted into the abdomen 30as a surgical target from the hole of the trocar 131 attached to theabdominal wall of the abdomen 30. The optical system of the endoscope111 applies light emitted from the light source apparatus 102 to theinside of the abdomen 30, and the imaging unit captures an image insidethe abdomen 30 as an intraoperative image.

The endoscope 111 supplies the intraoperative image to the CCU 101 viathe camera cable. This procedure displays the intraoperative imagecaptured by the endoscope 111 on the display apparatus 105, so as toenable the operator to perform treatment such as resection of anaffected site inside the abdomen 30 using the energy treatment tool 112while viewing the image inside the abdomen 30 in real time.

The energy treatment tool 112 includes an electric scalpel and the like.The energy treatment tool 112 is inserted into the abdomen 30 from thehole of the trocar 132 attached to the abdominal wall of the abdomen 30.The energy treatment tool 112 modifies or cuts the inside of the abdomen30 using electrical heat.

The forceps 135 are inserted into the abdomen 30 from the hole of thetrocar 134 attached to the abdominal wall of the abdomen 30. The forceps135 grip the inside of the abdomen 30. The endoscope 111, the energytreatment tool 112, and the forceps 135 are gripped by an operator, anassistant, a scope specialist, a robot, or the like.

The foot switch 121 receives operation by a foot of an operator, anassistant, or the like. The foot switch 121 supplies an operation signalindicating the received operation to the CCU 101 and the treatment toolapparatus 103. That is, the foot switch 121 controls the CCU 101, thetreatment tool apparatus 103, or the like, triggered by the operation bythe foot of the operator, the assistant, or the like.

The configuration of the endoscopic surgery system 10 is as describedabove. By using the endoscopic surgery system 10, the operator canexcise the affected site in the abdomen 30 without performing abdominalsurgery of cutting and opening the abdominal wall.

Hereinafter, a detailed configuration of the endoscopic surgery system10 in FIG. 1 will be described focusing on a configuration of the CCU101 and the endoscope 111. Moreover, processing in the CCU 101 and theendoscope 111 is performed following a processing mode determined inaccordance with surgical scenes, and the processing mode is determinedeither in accordance with an external signal or in accordance with scenerecognition. Accordingly, in the following description, determination ofthe processing mode using an external signal will be first described asa first embodiment, and thereafter, determination of the processing modeby scene recognition will be described as a second embodiment.

2. First Embodiment: Determination of Processing Mode Using ExternalSignal

(Exemplary Configuration of CCU and Endoscope)

FIG. 2 is a diagram illustrating a detailed exemplary configuration ofthe CCU 101 and the endoscope 111 according to the first embodiment.

The CCU 101 includes a processing mode determination unit 151 and animage combining unit 152. Moreover, the endoscope 111 includes a sensor161 and a sensor control unit 162.

The processing mode determination unit 151 determines a processing modecorresponding to the surgical scene in accordance with an externalsignal input from the outside of the CCU 101. The processing modedetermination unit 151 supplies the determined processing mode to theimage combining unit 152 and to the sensor control unit 162 of theendoscope 111.

Note that the external signal includes an operation signal correspondingto processing mode switching operation performed by an operator or anassistant, for example. For example, in a case where operation on aninput unit (user interface: UI) is received from the operator, theassistant, or the like, the operation signal is to be input to theprocessing mode determination unit 151 as an external signal.

Moreover, the external signal includes a light source switching signalsupplied from the light source apparatus 102 (FIG. 1). For example, in acase where the light source is switched in accordance with observationsuch as narrowband imaging (NBI), fluorescence observation in responseto operation by the operator, or the like, a switching signal is to beinput as an external signal to the processing mode determination unit151.

Furthermore, the external signal includes a signal indicating poweron-off of the energy treatment tool 112 such as an electric scalpelconnected to the treatment tool apparatus 103 (FIG. 1) via a cable. Forexample, in a case where power of the energy treatment tool 112 such asan electric scalpel is turned on or off, a signal indicating the poweron/off is to be input to the processing mode determination unit 151 asan external signal.

Moreover, the processing mode determination unit 151 stores a table(hereinafter referred to as a correspondence table) associating asurgical scene with a processing mode. In a case where an externalsignal is input from the outside of the CCU 101, the processing modedetermination unit 151 determines a processing mode corresponding to asurgical scene specified by the external signal with reference to thecorrespondence table.

Furthermore, although details will be described below with reference toFIG. 4, FIG. 6 or the like, the correspondence table associates asurgical scene such as narrowband imaging (NBI), fluorescenceobservation, suturing, and stripping with a processing mode such as ahigh definition mode and a high frame rate (HFR) mode. As the processingmodes, a mode according to processing details of image processing suchas the presence or absence of pixel shift processing, a frame rate(unit: frame per second (fps)) of an output image (intraoperativeimage), resolution of the output image (intraoperative image) is set.

The sensor control unit 162 controls the sensor 161 on the basis of theprocessing mode supplied from the processing mode determination unit151. Moreover, the sensor control unit 162 supplies an image signalsupplied from the sensor 161 to the image combining unit 152 of the CCU101 via the camera cable.

Examples of the sensor 161 include a solid-state imaging element(imaging element) such as a complementary metal oxide semiconductor(CMOS) image sensor and a charge coupled device (CCD) image sensor.

The sensor 161 includes, for example, an image array unit including aplurality of two-dimensionally arranged pixels having photoelectricconversion elements (photodiodes) that receives incident light from alens, a peripheral circuit unit that performs pixel drive,analog/digital (A/D) conversion, and the like. The sensor 161 usesphotoelectric conversion elements to photoelectrically convert lightthat has passed through the lens and focused on a light receivingsurface, applies predetermined signal processing to covert the lightinto an image signal and supplies the obtained image signal (imagesignal of in-vivo image) to the sensor control unit 162.

Note that it is possible to use as the imaging system of the endoscope111, a single-chip sensor 161 (hereinafter also described as a sensor161-1) or a three-chip sensor 161 (hereinafter also described as asensor 161-3). The single-chip method is a method using a single sensor(solid-state imaging element) having each of pixels of RGB as the sensor161-1. In addition, the three-chip method is a method using threesensors (solid-state imaging elements) having pixels each having each ofcomponents of RGB, as the sensor 161-3.

Moreover, the imaging system of the endoscope 111 can employ a techniqueof pixel shift processing. With this pixel shift processing technique,it is possible to obtain a high definition image.

Hereinafter, a method of temporally shifting the single-chip sensor161-1 (single sensor) in a case where the single-chip sensor 161-1 isapplied in the imaging system of the endoscope 111 will be referred toas a single-chip pixel shift method. Moreover, in a case where thethree-chip sensor 161-3 is applied in the imaging system of theendoscope 111, a method of generating optical phase shifts by thethree-chip sensor 161-3 (three sensors corresponding to the RGBcomponents) and simultaneously imaging each of planes in an RGB phaseshifted state using the three-chip sensor 161-3 (three sensorscorresponding to the RGB components) will be referred to as a three-chippixel shift method.

For example, in the case where the single-chip sensor 161-1 is appliedin the imaging system of the endoscope 111, the sensor control unit 162controls the single-chip pixel shift in the single-chip sensor 161-1 inaccordance with the processing mode supplied from the processing modedetermination unit 151. Herein, in the case of performing single-chippixel shift in accordance with the processing mode, the sensor controlunit 162 controls the driving unit such as a piezoelectric drivingapparatus to move the single-chip sensor 161-1 in units of pixels.

Moreover, for example, in the case of using the three-chip sensor 161-3in the imaging system of the endoscope 111, the sensor control unit 162performs shutter control or the like of the endoscope 111 in accordancewith the processing mode.

An image signal is supplied from the sensor control unit 162 of theendoscope 111 to the image combining unit 152 via the camera cable. Onthe basis of the processing mode supplied from the processing modedetermination unit 151, the image combining unit 152 performspredetermined image processing on the image signal from the sensorcontrol unit 162, and outputs the processed signal as an output image(intraoperative image).

For example, in the case where the single-chip sensor 161-1 is appliedin the imaging system of the endoscope 111, the image combining unit 152performs predetermined image processing corresponding to the processingmode, such as processing of adding an image signal output from thesingle-chip sensor 161-1 (single sensor). Moreover, for example, in acase where the three-chip sensor 161-3 is applied in the imaging systemof the endoscope 111, the image combining unit 152 performspredetermined image processing corresponding to the processing mode,such as processing of combining an image signal output from thethree-chip sensor 161-3 (three sensors corresponding to RGB components).

In the endoscopic surgery system 10 (FIG. 1), the CCU 101 and theendoscope 111 are configured as described above. Next, the imagingsystem of the endoscope 111 in a case where the single-chip sensor 161-1is applied and in a case where the three-chip sensor 161-3 is appliedwill be sequentially described in this order.

(1) Case where Single-Chip Sensor is Applied

(Outline of Single-Chip Pixel Shift Method)

FIG. 3 is a diagram illustrating an outline of a single-chip pixel shiftmethod.

FIG. 3 illustrates two examples of a single-plate pixel shift method oftemporally shifting a single solid-state imaging element in A of FIG. 3and B of FIG. 3. Herein, 4×4 pixels are illustrated as representativeexamples, among a plurality of pixels two-dimensionally arranged on thelight receiving surface of the single-chip sensor 161-1 (solid-stateimaging element).

Moreover, an R pixel capable of obtaining a red (R) image signal isdescribed as “R”. Similarly, a G pixel capable of obtaining a green (G)image signal is described as “G”, while a B pixel capable of obtaining ablue (B) image signal is described as “B”.

(A) Single-Chip Pixel Shift Method A

In A of FIG. 3, the numerals 1 to 4 described in individual pixelsrepresent the number of times of imaging. For example, image signalscorresponding to four pixels with patterns in odd rows and odd columns,that is, pixel G1, pixel R1, pixel B1, and pixel G1, are to be obtainedby first imaging. Note that by focusing on these four pixels, a Bayerpattern is generated in which the pixels G1 are arranged in acheckerboard pattern, with the pixels R1 and the pixels B1 alternatelyarranged in a row in the remaining portion.

After the first imaging, the single-chip sensor 161-1 is moved by onepixel in the right direction (row direction) in the figure, wherebyimage signals corresponding to four pixels in the odd rows and the evencolumns arranged in the Bayer pattern, that is, pixel G2, pixel R2,pixel B2, and pixel G2, are obtained by second imaging.

Moreover, after the second imaging, the single-chip sensor 161-1 ismoved by one pixel in the left diagonal downward direction in thefigure, whereby image signals corresponding to four pixels in the evenrows and the odd columns arranged in the Bayer pattern, that is, pixelG3, pixel R3, pixel B3, and pixel G3, are obtained by third imaging.

Subsequently, after the third imaging, the single-chip sensor 161-1 ismoved by one pixel in the right direction (row direction) in the figure,whereby image signals corresponding to four pixels in the even rows andthe even columns arranged in the Bayer pattern, that is, pixels G4,pixel R4, pixel B4, and pixel G4, are obtained by fourth imaging.

In this manner, by shifting the single-chip sensor 161-1 (solid-stateimaging element) in units of pixels to repeat four shots of imaging intotal at each of the positions in the single-chip pixel shift method Aillustrated in A of FIG. 3, image signals corresponding to four shots ofimaging can be obtained from each of the pixels, making it possible toobtain quadrupled output image resolution.

(B) Single-Chip Pixel Shift Method B

In B of FIG. 3, similarly to A of FIG. 3, the numerals 1 to 4 describedin individual pixels represent the number of times of imaging. Forexample, image signals corresponding to four pixels in an upper leftcorner arranged in the Bayer pattern, that is, pixel G1, pixel R1, pixelB1, and pixel G1 are to be obtained by first imaging.

After the first imaging, the single-chip sensor 161-1 is moved by twopixels in the upper right direction (row direction) in the figure,whereby image signals corresponding to four pixels in an upper rightcorner arranged in the Bayer pattern, that is, pixel G2, pixel R2, pixelB2, and pixel G2, are obtained by second imaging.

Moreover, after the second imaging, the single-chip sensor 161-1 ismoved by two pixels in the left diagonal downward direction in thefigure, whereby image signals corresponding to four pixels in the lowerleft corner arranged in the Bayer pattern, that is, pixels G3, pixel R3,pixel B3, and pixel G3, are obtained by third imaging.

Subsequently, after the third imaging, the single-chip sensor 161-1 ismoved by two pixels in the right direction (row direction) in thefigure, whereby image signals corresponding to four pixels in the lowerright corner arranged in the Bayer pattern, that is, pixel G4, pixel R4,pixel B4, and pixel G4, are obtained by fourth imaging.

In this manner, in the single-chip pixel shift method B illustrated in Bof FIG. 3, by shifting the single-chip sensor 161-1 (solid-state imagingelement) in units of pixels to repeat four shots of imaging in total ateach of the positions, image signals corresponding to four shots ofimaging can be obtained from each of the pixels, making it possible toobtain quadrupled output image resolution.

Note that the single-chip pixel shift method A in A of FIG. 3 and thesingle-chip pixel shift method B in B of FIG. 3 are exemplarysingle-chip pixel shift methods of the single-chip sensor 161-1, and itis allowable to employ other method to perform single-chip pixel shiftof the single-chip sensor 161-1.

(Correspondence Table of Single-Chip Sensor)

FIG. 4 is a diagram illustrating an exemplary correspondence table in acase where the single-chip sensor 161-1 is applied in the imaging systemof the endoscope 111.

The correspondence table in FIG. 4 is a table associating surgicalscenes and processing modes in a case where the single-chip sensor 161-1is applied. Note that there are four types of processing modes in a casewhere the single-chip sensor 161-1 is applied, that is, a normal mode, ahigh definition mode, a high frame rate (HFR) mode, and a noisereduction (NR) mode. Note that the single-chip pixel shift illustratedin FIG. 3 is performed in the high definition mode among theseprocessing modes.

The normal mode is a mode in which single-chip pixel shift is notperformed and an image (image data) captured at 60 fps is output at 60fps. For example, in A of FIG. 3 (or B of FIG. 3), imaging is performedselectively at each of the pixel positions at which the numeral 1 isdescribed, that is, at pixel G1, pixel R1, pixel B1, and pixel G1, andan output image obtained from image signals output from these pixels isoutput at 60 fps.

The high definition mode is a mode in which single-pixel shift isperformed and a high definition image (image data) with quadrupledresolution is output at 60 fps. For example, in A in FIG. 3 (or B inFIG. 3), the single-chip sensor 161-1 is shifted in units of pixels in1/60 seconds at the position of pixel in which 1 to 4 numerals aredescribed, and four shots of imaging in total is repeated at each of thepositions. With this imaging, a high definition image having quadrupledresolution compared with the normal mode or the like is generated andoutput at 60 fps as an output image. That is, this high definition modeis applied to perform the single-chip pixel shift illustrated in FIG. 3to output an output image with higher resolution compared with othermodes.

Furthermore, the high definition images obtained in the high definitionmode include not merely high definition (HD) video images, but also, forexample, ultra high definition (UHD) video images having 4K resolution(for example 3840 in width×2160 in height) and 8K resolution (forexample 7680 in width×4320 in height).

The HFR mode is a mode in which single-chip pixel shift is not performedand the image (image data) is obtained by four shots of imaging during1/60 second and then output at 240 fps. For example, in A of FIG. 3 (orB of FIG. 3), four shots of imaging are performed selectively at each ofthe pixel positions at which the numeral 1 is described, that is, atpixel G1, pixel R1, pixel B1, and pixel G1 during 1/60 second, and anoutput image obtained from the image signal output from these pixels isoutput at 240 fps. That is, in this HFR mode, the output image can beoutput at a higher frame rate than in other modes.

The NR mode is a mode in which single-chip pixel shift is not performedand the four images (image data) are obtained by four shots of imagingduring 1/60 second, added and then output at 60 fps. For example, in Aof FIG. 3 (or B of FIG. 3), four shots of imaging are performedselectively at each of the pixel positions at which the numeral 1 isdescribed, that is, at pixel G1, pixel R1, pixel B1, and pixel G1 during1/60 second, and the output image obtained by adding the image signalsoutput from these pixels is output at 60 fps. That is, in this NR mode,the output image with less noise compared with other modes can beoutput.

In the correspondence table of FIG. 4, each of surgical scenes isassociated with each of these processing modes. That is, each of thescenes of the narrowband imaging (NBI) and fluorescence observation isassociated with the processing mode of either the high definition modeor the NR mode. In addition, the suturing scene is associated with theprocessing mode of the high definition mode.

The scope movement scene is associated with the processing mode of theHFR mode. Moreover, the careful stripping scene is associated with theprocessing mode of the high definition mode, while the normal strippingscene is associated with the processing mode of the HFR mode.

Note that the normal mode is associated with scenes other than theabove-described surgical scenes, that is, scenes other than narrowbandimaging, fluorescence observation, suturing, scope movement, carefulstripping, and normal stripping scenes.

In this manner, in the correspondence table of FIG. 4, the highdefinition mode and the NR mode are applied to provide high definitionimages and images with less noise for the scenes where the scopemovement is relatively small and the operator wishes to watch in moredetail, such as narrowband imaging (NBI), fluorescence observation,careful stripping, and suturing. Moreover, for example, the narrowbandimaging (NBI) performs observation with limited amount of light, thatis, in a dark condition, leading to a phenomenon of increasing noise inan output image. In this case, with a setting to select (determine) theNR mode as the processing mode, it is possible to remove the noiseincluded in the output image.

Moreover, in the correspondence table of FIG. 4, for example, the HFRmode is applied to provide an image (high frame rate image) enabling themotion to appear smoother for a scene including a large motion of ascope such as scope movement or for a normal stripping scene. With thismode, for example, fatigue of the operator or assistant can be reduced.

In this manner, it is not always a good choice to display a highdefinition image (video image) and there is a case where an image (videoimage) with a higher frame rate or an image (video image) with lessnoise is more suitable depending on the surgical scene. Therefore,according to the present technology, an appropriate processing modecorresponding to the surgical scene is to be determined by thecorrespondence table of FIG. 4.

Note that in a case where the single-chip sensor 161-1 is applied, theprocessing mode determination unit 151 of the CCU 101 stores thecorrespondence table (FIG. 4) beforehand so as to determine theprocessing mode corresponding to the surgical scene specified by theexternal signal.

The correspondence table in a case where the single-chip sensor 161-1 isapplied in the imaging system of the endoscope 111 has been describedabove.

(2) Case where Three-Chip Sensor is Applied

(Outline of Three-Chip Pixel Shift Method)

FIG. 5 is a diagram illustrating an outline of the three-chip pixelshift method.

As illustrated in A of FIG. 5, the three-chip sensor 161-3 functionssuch that light passing through the lens is divided into an R component,a G component, and a B component by a prism, so as to be received by asensor 161-R, a sensor 161-G, and a sensor 161-B, respectively,corresponding to each of the components, allowing an image signal ofeach of the components to be output from each of the sensors.

That is, A of FIG. 5 illustrates an exemplary configuration as thethree-chip sensor 161-3 including the sensor 161-R for receiving Rcomponent light, the sensor 161-G for receiving G component light, andthe sensor 161-B for receiving the B component light. In this manner,the three-chip sensor 161 includes dedicated sensors (sensor 161-R,sensor 161-G, and sensor 161-B) applicable for each of the three primarycolors of RGB, leading to an advantage of achieving excellent colorreproducibility and resolution.

In addition, as illustrated in B of FIG. 5, in the case of implementingthe three-chip pixel shift method by the three-chip sensor 161-3, forexample, the sensor 161-R is shifted by ½ pixel downward (in columndirection) in the drawing and the sensor 161-B is shifted by ½ pixelrightward (in row direction) in the drawing with respect to the sensor161-G as a reference.

That is, as illustrated in B of FIG. 5, the sensor 161-R and the sensor161-B are shifted with respect to the sensor 161-G to generate anoptical phase shift, and individual planes are simultaneously imaged bythe sensor 161-R, the sensor 161-G, and the sensor 161-B in a statewhere the phases of RGB are shifted, whereby a high definition image canbe obtained. Note that in the following description, it is assumed thatthree-chip pixel shift is performed in the three-chip sensor 161-3 asillustrated in B of FIG. 5. In addition, while the three-chip sensor161-3 will be described as an example in the following, it is alsoallowable to use, for example, three or more sensors (solid-stateimaging elements) in the case of using an infrared (IR) component inaddition to the RGB components.

(Correspondence Table of Three-Chip Sensor)

FIG. 6 is a diagram illustrating an exemplary correspondence table in acase where the three-chip sensor 161-3 is applied in the imaging systemof the endoscope 111.

The correspondence table in FIG. 6 is a table associating surgicalscenes and processing modes in a case where the three-chip sensor 161-3is applied. Note that the processing mode in a case where the three-chipsensor 161-3 is applied includes two types of modes, namely, the highdefinition mode and the HFR mode.

The high definition mode is a mode of imaging each of RGB planes at asame timing, generating a high definition image from the RGB planeimages, and outputting the image as an output image at 60 fps.

Note that the method for generating a high definition image from each ofRGB planes with shifted phases, for example, include a method ofgenerating a pixel at a position where no pixel is present usinginterpolation for each of the plane by a linear filter (Linear Filter)or a general bicubic filter (Bi-Cubic Filter). Note that the techniquefor generating the high definition image exemplified here isillustrative, and other known methods can be used. For example, JapanesePatent Laid-Open No. 2000-13670 discloses a technique using a linearfilter.

Note that the high definition image obtained in this high definitionmode includes, for example, UHD video images achieving 4K resolution and8K resolution as well as HD video images. That is, with the highdefinition mode, it is possible to output an output image with higherresolution than in the HFR mode. Furthermore, details of the highdefinition mode will be described below with reference to FIG. 7.

The HFR mode is a mode of combining images of RBG planes having closeimaging time among the RBG plane images captured with shifted imagingtimings as a result of imaging of RGB planes with different timings andoutputting the combined image as an output image at 120 fps. That is, inthis HFR mode, it is possible to output an output image at a higherframe rate than in the high definition mode. Furthermore, details of theHFR mode will be described below with reference to FIGS. 7 and 8.

In the correspondence table of FIG. 6, individual surgical scenes areassociated with these processing modes. That is, the narrowband imaging(NBI) and fluorescence observation scenes are associated with theprocessing mode of the high definition mode. In addition, the suturingscene is associated with the processing mode of the high definitionmode.

The scope movement scene is associated with the processing mode of theHFR mode. Moreover, the careful stripping scene is associated with theprocessing mode of the high definition mode, while the normal strippingscene is associated with the processing mode of the HFR mode.

Note that the surgical scenes other than the above, that is, thesurgical scenes not associated with the high definition mode or the HFRmode can be associated with the processing mode of the normal mode. Inthe normal mode, an output image is output without undergoing anyspecial processing.

In this manner, in the correspondence table of FIG. 6, high definitionmode is applied to provide high definition images for the scenes wherethe scope movement is relatively small and the operator wishes to watchin more detail, such as narrowband imaging (NBI), fluorescenceobservation, careful stripping, and suturing.

Moreover, in the correspondence table of FIG. 6, for example, the HFRmode is applied to provide an image (high frame rate image) enabling themotion to appear smoother for a scene including a large motion of ascope such as scope movement or for a normal stripping scene.

Furthermore, in comparison between the correspondence table (FIG. 4) forapplication of the single-chip sensor 161-1 and the correspondence table(FIG. 6) for application of the three-chip sensor 161-3, there is adifference in the correspondence table (FIG. 6) for application of thethree-chip sensor 161-3 that the NR mode is not included in theprocessing mode. This is because it is difficult to obtain a pluralityof images without shifting pixels due to the structure of the three-chipsensor 161-3.

In this manner, it is not always a good choice to display a highdefinition image (video image) and there is a case where an image (videoimage) with a higher frame rate is more suitable depending on thesurgical scene. Therefore, according to the present technology, anappropriate processing mode corresponding to the surgical scene is to bedetermined by the correspondence table of FIG. 6.

Note that in a case where the three-chip sensor 161-3 is applied, theprocessing mode determination unit 151 of the CCU 101 stores thecorrespondence table (FIG. 6) beforehand so as to determine a processingmode corresponding to the surgical scene specified by the externalsignal.

The correspondence table used in a case where the three-chip sensor161-3 is applied in the imaging system of the endoscope 111 has beendescribed above.

(RGB Signal Output Timing for Each of Processing Modes)

FIG. 7 is a timing chart illustrating RGB signal output timings for eachof the processing modes. In FIG. 7, the time direction is a directionfrom the left side to the right side in the figure.

A of FIG. 7 is a timing chart illustrating RGB signal output timings forthe HFR mode. Moreover, B of FIG. 7 is a timing chart illustrating RGBsignal output timings for the high definition mode.

Moreover, in the timing chart of A of FIG. 7 and B of FIG. 7, an imagesignal corresponding to the R component output from the sensor 161-R isdescribed as an “R signal”. Similarly, an image signal corresponding tothe G component output from the sensor 161-G is described as a “Gsignal”, while an image signal corresponding to the B component outputfrom the sensor 161-B is described as a “B signal”.

That is, in the high definition mode timing chart in B of FIG. 7illustrating the outputs of the R signal, the G signal, and the B signalin chronological order, the sensor 161-R, the sensor 161-G, and thesensor 161-B are capable of outputting the R signal, the G signal, andthe B signal, respectively, at a same timing because the shutter is in areleased state in the three-chip sensor 161-3 in the endoscope 111.

With this configuration, the high definition mode enables imaging ofeach of the RGB planes at a same timing, generating a high definitionimage from the obtained RGB plane images, and outputting the image at 60fps as an output image. For example, focusing on the period from time t2to time t6 in the timing chart of B of FIG. 7, a high definition image(output image) is to be output at the timing of each of time t2, timet4, and time t6.

In contrast, in the timing chart of the HFR mode in A of FIG. 7illustrating the outputs of the R signal, the G signal, and the B signalin chronological order, a shutter of the endoscope 111 (three-chipsensor 161-3) is controlled to perform imaging with halved exposure timeas compared with the case of the high definition mode, and in addition,at shifted timing for each of the RGB planes.

Furthermore, the RGB plane images captured at close imaging times are tobe combined with each other among the RGB plane images captured withshifted imaging timings. Specifically, for example, focusing on theperiod from time t2 to time t6 in the timing chart of A in FIG. 7, forexample, the G signal obtained in the period from time t1 to time t2 isused, at time t2, so as to combine the R signal and the B signaltemporally close to the G signal.

Similarly, at time t3, for example, the B signal obtained in the periodfrom time t2 to time t3 is used to combine the R signal and the G signaltemporally close to the B signal. Moreover, at time t4, for example, theR signal obtained in the period from time t3 to time t4 is used tocombine the G signal and the B signal temporally close to the R signal.

Furthermore, at time t5, for example, the G signal obtained in theperiod from time t4 to time t5 is used to combine the R signal and the Bsignal temporally close to the G signal. Moreover, at time t6, forexample, the B signal obtained in the period from time t5 to time t6 isused to combine the R signal and the G signal temporally close to the Bsignal.

In this manner, the HFR mode is capable of imaging with the halvedexposure time compared with the high definition mode to capture each ofthe RGB planes at different timings, combining RGB plane images havingclose imaging times among the RGB plane images captured with the shiftedimaging timings, and outputting the image at 120 fps. For example,focusing on the period from time t2 to time t6 in the timing chart in Aof FIG. 7, a high frame rate image (output image) is output at a timingof each of the time from time t2 to time t6.

Note that, comparing the timing chart of the HFR mode in A of FIG. 7with the timing chart of the high definition mode in B of FIG. 7, theoutput image is output at the time of time t2, time t4, and time t6among the period from time t2 to time t6 in the high definition mode,while the output image is output at the timing of each of the timepoints from time t2 to time t6 in the HFR mode. That is, the frame rate(for example, 120 fps) of the output image in the HFR mode is twice theframe rate (for example, 60 fps) of the output image in the highdefinition mode.

Note that while the exemplary timing chart in FIG. 7 uses the exposuretime in the HFR mode being halved compared with the case in the highdefinition mode, the exposure time can be further reduced to ⅓, ¼, orthe like, to further increase the frame rate of the output image in theHFR mode.

Meanwhile, since images to be combined (R signal, G signal, and Bsignal) are not images captured at the same timing at the time ofcombining the RGB plane images having closer imaging times (temporallyshifted captured images), there is a case where a subject included inthe image moves. Herein, in order to cope with such a case, the imagecombining unit 152 (FIG. 2) of the CCU 101 has a configuration asillustrated in FIG. 8.

In FIG. 8, the image combining unit 152 corresponding to the HFR modeincludes a buffer 181-R, a buffer 181-G, a buffer 181-B, a misalignmentdetection unit 182, and a misalignment correction unit 183.

The buffer 181-R is a buffer to hold the R signal output from the sensor161-R of the endoscope 111. The buffer 181-G is a buffer to hold the Gsignal output from the sensor 161-G of the endoscope 111. The buffer181-B is a buffer to hold the B signal output from the sensor 161-B ofthe endoscope 111.

For example, in a case where the G signal obtained in the period fromtime t1 to time t2 is used at time t2 in the timing chart of the HFRmode in A of FIG. 7 described above, the G signal is held in the buffer181-G, and the R signal and the B signal temporally close to the Gsignal are held in the buffer 181-R and the buffer 181-B, respectively.

The R signal, the G signal, and the B signal respectively held in thebuffer 181-R, the buffer 181-G, and the buffer 181-B are read by themisalignment detection unit 182 and the misalignment correction unit183.

The misalignment detection unit 182 applies block matching, mutualcorrelation, or the like, to the R signal, the G signal, and the Bsignal respectively read from the buffer 181-R, the buffer 181-G, andthe buffer 181-B, thereby detecting the misalignment amount of each ofthe signals for each of the pixels. The misalignment detection unit 182supplies the misalignment amount of each of the detected signals to themisalignment correction unit 183.

The misalignment correction unit 183 receives inputs of the R signal,the G signal, and the B signal respectively read from the buffer 181-R,the buffer 181-G, and the buffer 181-B, as well as inputs of themisalignment amounts of the individual signals synchronized with thesesignals, from the misalignment detection unit 182.

On the basis of the misalignment amount of each of the signals from themisalignment detection unit 182, the misalignment correction unit 183performs misalignment correction for each of the pixels of the R signal,the G signal, and the B signal respectively read from the buffer 181-R,the buffer 181-G, and the buffer 181-B. The misalignment correction unit183 collectively outputs the R signal, G signal, and B signal alignedwith each other by misalignment correction, as an RGB signal (outputimage).

In this manner, since the images to be combined (R signal, G signal, andB signal) are not images captured at the same timing at the time ofcombining the RGB plane images with close imaging times, there is apossibility that the subject included in the image moves. Therefore, theimage combining unit 152 (FIG. 8) of the CCU 101 processes the R signal,the G signal, and B signal, buffered in each of the buffers 181, usingthe misalignment detection unit 182 and the misalignment correction unit183 so as to perform correction to achieve alignment of the position ofthe subject that has moved. With this configuration, it is possible tooutput, in the HFR mode, an output image at a timing of 120 fps which istwice the high definition mode, for example.

(Image Combining Processing Flow)

Next, an image combining processing flow according to the firstembodiment will be described with reference to the flowchart in FIG. 9.

In step S101, the processing mode determination unit 151 receives anexternal signal input from the outside of the CCU 101.

Note that the external signal to be provided includes, for example, anoperation signal corresponding to processing mode switching operation byan operator or the like, a light source switching signal supplied fromthe light source apparatus 102 (FIG. 1), or a signal indicating poweron-off of the energy treatment tool 112.

In step S102, the processing mode determination unit 151 determines aprocessing mode corresponding to the surgical scene specified by theexternal signal received in the processing of step S101 with referenceto the correspondence table stored beforehand.

For example, in a case where the single-chip sensor 161-1 is applied inthe imaging system of the endoscope 111, the processing modedetermination unit 151 determines the processing mode corresponding tothe surgical scene with reference to the correspondence table in FIG. 4.Moreover, in a case where the three-chip sensor 161-3 is applied in theimaging system of the endoscope 111, for example, the processing modedetermination unit 151 determines the processing mode corresponding tothe surgical scene with reference to the correspondence table in FIG. 6.

In step S103, the sensor control unit 162 controls the sensor 161 on thebasis of the processing mode determined by the processing in step S102.The sensor 161 outputs an image signal (image signal of an in-vivoimage) under the control of the sensor control unit 162.

For example, in a case where the single-chip sensor 161-1 is applied,the sensor control unit 162 controls the single-chip pixel shiftcorresponding to the processing mode. More specifically, the sensorcontrol unit 162 turns on the pixel shift processing by the single-chipsensor 161-1 in a case where the processing mode is the high definitionmode, while the unit turns off the pixel shift processing by thesingle-chip sensor 161-1 in a case where the processing mode is set tothe mode other than the high definition mode.

Moreover, in a case where the three-chip sensor 161-3 is applied, forexample, the sensor control unit 162 performs shutter control of theendoscope 111 in accordance with the processing mode. More specifically,in a case where the processing mode is the HFR mode, the sensor controlunit 162 controls to halve the exposure time as compared with the caseof the high definition mode and controls an imaging timing shift foreach of the RGB planes by shutter speeds so as to implement imaging atshifted timing for each of the RGB planes.

In step S104, the image combining unit 152 obtains an image signaloutput from the endoscope 111 (the sensor 161 thereof) via a cameracable.

In step S105, the image combining unit 152 combines (processes) theimage signal obtained by the processing in step S104 on the basis of theprocessing mode determined by the processing in step S102. The outputimage obtained by the processing of step S105 is output at apredetermined frame rate.

For example, in a case where the single-chip sensor 161-1 is applied,the image combining unit 152 performs predetermined image processingcorresponding to the processing mode. More specifically, in a case wherethe processing mode is the NR mode, for example, the image combiningunit 152 performs image processing of adding image signals obtained byfour shots of imaging.

Moreover, for example, in a case where the three-chip sensor 161-3 isapplied, the image combining unit 152 performs predetermined imageprocessing corresponding to the processing mode. More specifically, in acase where the processing mode is the high definition mode, for example,the image combining unit 152 performs image processing of combining theR signal, the G signal, and the B signal. Moreover, as described above,in a case where the processing mode is the HFR mode, the image combiningunit 152 uses the configuration illustrated in FIG. 8 to perform theprocessing described with reference to FIG. 8.

In step S106, it is determined whether to finish the processing. In acase where it is determined in step S106 that the processing is not tobe finished, the processing returns to step S101 to repeat subsequentprocessing. In addition, in a case where it is determined in step S106that the processing is to be finished, the image combining processing ofthe first embodiment in FIG. 9 is finished.

The image combining processing flow according to the first embodimenthas been described above. In the image combining processing, theprocessing mode determination unit 151 refers to the correspondencetable (for example, the correspondence table in FIG. 4 or FIG. 6) todetermine the processing mode corresponding to the surgical scenespecified by the external signal, and then, the sensor control unit 162and the image combining unit 152 perform processing corresponding to theprocessing mode determined by the processing mode determination unit151. That is, an appropriate processing mode corresponding to thesurgical scene is determined by the correspondence table (for example,the correspondence table in FIG. 4 or FIG. 6) and the processingcorresponding to the processing mode is performed, making it possible toprovide an optimal video image to the operator in accordance with thesurgical scenes.

3. Second Embodiment: Determination of Processing Mode by SceneRecognition

Meanwhile, the first embodiment describes a case where a surgical sceneis specified by an external signal input from the outside of the CCU 101to determine the processing mode corresponding to the surgical scenewith reference to the correspondence table (for example, thecorrespondence table in FIG. 4 or FIG. 6). As a method for specifyingthe surgical scene, it is allowable to use, for example, a method inwhich a scene recognition processing is performed on an output image(intraoperative image) to specify the surgical scene in accordance witha scene recognition result, other than the method using the externalsignal.

Accordingly, a method of specifying a surgical scene by scenerecognition processing on an output image (intraoperative image) will bedescribed as a second embodiment.

(Exemplary Configuration of CCU and Endoscope)

FIG. 10 is a diagram illustrating a detailed exemplary configuration ofthe CCU 101 and the endoscope 111 according to the second embodiment.Note that in the CCU 101 and the endoscope 111 in FIG. 10, the samereference numerals are given to the blocks corresponding to the CCU 101and the endoscope 111 in FIG. 2, and repetitive description thereof willbe omitted as appropriate.

Specifically, the CCU 101 in FIG. 10 is different from the CCU 101 inFIG. 2 in that a scene recognition unit 153 is provided at a stagefollowing the image combining unit 152, and that a scene recognitionresult obtained by the scene recognition unit 153 is fed back to theprocessing mode determination unit 151. Note that the endoscope 111 inFIG. 10 and the endoscope 111 in FIG. 2 have a same configuration.

In the CCU 101 of FIG. 10, an output image (image signal) output fromthe image combining unit 152 is input to the scene recognition unit 153.The scene recognition unit 153 performs scene recognition processing onthe output image (intraoperative image) from the image combining unit152. By the scene recognition processing, a predetermined scene isrecognized from the output image (intraoperative image), and thesurgical scene is automatically discriminated. Subsequently, a scenerecognition result (surgical scene discrimination result) obtained inthe scene recognition processing is fed back to the processing modedetermination unit 151.

Note that, for example, the scene recognition result (surgical scenediscrimination result) fed back from the scene recognition unit 153 tothe processing mode determination unit 151 can include the followinginformation.

That is, for example, it is possible to include a surgical scenediscrimination result as being narrowband imaging (NBI), fluorescenceobservation, or normal observation, corresponding to distributionanalysis of RGB histograms on an output image (intraoperative image).Moreover, for example, it is possible to include a surgical scenediscrimination result as being a suture corresponding to detection of athread or a suture needle included in an output image (intraoperativeimage).

Furthermore, it is possible to include, for example, a surgical scenediscrimination result as being a scope movement corresponding todetection of a frame difference or motion vector of an output image(intraoperative image). Moreover, for example, it is possible to includea surgical scene discrimination result as being stripping (normalstripping or careful stripping) corresponding to detection of theforceps 135 included in the output image (intraoperative image) andmotion detection of the forceps 135.

Note that the surgical scene discrimination results listed herein areillustrative, and for example, it is allowable to cause another scenerecognition result obtained using known image analysis processing to befed back to the processing mode determination unit 151.

The scene recognition result (discrimination result of the surgicalscene) corresponding to the output image (intraoperative image) is inputfrom the scene recognition unit 153 to the processing mode determinationunit 151 at a predetermined timing.

In a case where the scene recognition result has been input from thescene recognition unit 153, the processing mode determination unit 151determines the processing mode corresponding to a surgical scenespecified by the scene recognition result (surgical scene discriminationresult) with reference to the correspondence table stored beforehand(for example, the correspondence table in FIG. 4 or FIG. 6). Theprocessing mode determination unit 151 supplies the determinedprocessing mode to the image combining unit 152 and to the sensorcontrol unit 162 of the endoscope 111.

The sensor control unit 162 controls the sensor 161 on the basis of theprocessing mode supplied from the processing mode determination unit151. Moreover, on the basis of the processing mode supplied from theprocessing mode determination unit 151, the image combining unit 152performs predetermined image processing on an image signal supplied fromthe endoscope 111 (the sensor control unit 162 thereof) via a cameracable.

(Image Combining Processing Flow)

Next, an image combining processing flow according to the secondembodiment will be described with reference to the flowchart in FIG. 11.

In step S151, it is determined whether processing of the endoscopicsurgery system 10 has been started. In a case where it is determined instep S151 that the processing has been started, the processing proceedsto step S152.

In step S152, the processing mode determination unit 151 determines aninitial processing mode, and supplies the determined initial processingmode to the image combining unit 152 and the sensor control unit 162.

That is, for example, the initial processing mode set beforehand is tobe determined because the scene recognition unit 153 cannot obtain anoutput image (intraoperative image) from the image combining unit 152immediately after the processing is started in the endoscopic surgerysystem 10 (the CCU 101 thereof), and that the scene recognition result(surgical scene discrimination result) cannot be fed back to theprocessing mode determination unit 151.

In another case where it is determined in step S151 that the processinghas not been started, the processing proceeds to step S153. In stepS153, the scene recognition unit 153 performs scene recognitionprocessing on an output image (intraoperative image) corresponding to animage signal from the image combining unit 152.

With this processing in step S153, a predetermined scene from the outputimage (intraoperative image) is recognized and the surgical scene isautomatically discriminated. Subsequently, the scene recognition result(surgical scene discrimination result) obtained in the processing ofstep S153 is fed back to the processing mode determination unit 151.

In step S154, the processing mode determination unit 151 determines aprocessing mode corresponding to the surgical scene specified by thescene recognition result (surgical scene discrimination result) obtainedin the processing of step S153, with reference to the correspondencetable stored beforehand.

For example, in a case where the single-chip sensor 161-1 is applied inthe imaging system of the endoscope 111, the processing modedetermination unit 151 determines the processing mode corresponding tothe surgical scene discrimination result obtained as a feedback, withreference to the correspondence table in FIG. 4. Moreover, for example,in a case where the three-chip sensor 161-3 is applied in the imagingsystem of the endoscope 111, the processing mode determination unit 151determines the processing mode corresponding to the surgical scenediscrimination result obtained as a feedback, with reference to thecorrespondence table in FIG. 6.

In steps S155 to S157, the sensor control unit 162 controls the sensor161 on the basis of the processing mode determined by the processing instep S154, similarly to the steps S103 to S105 of FIG. 9, and the imagesignals are combined (processed) by the image combining unit 152 on thebasis of the processing mode determined by the processing in step S154.The output image obtained in the processing of step S157 is output at apredetermined frame rate.

In step S158, it is determined whether to finish the processing. In acase where it is determined in step S158 that the processing is not tobe finished, the processing returns to step S151 to repeat thesubsequent processing. In addition, in a case where it is determined instep S158 that the processing is to be finished, the image combiningprocessing of the second embodiment in FIG. 11 is finished.

The image combining processing flow according to the second embodimenthas been described above. In this image combining processing, theprocessing mode determination unit 151 refers to the correspondencetable (for example, the correspondence table in FIG. 4 or FIG. 6) todetermine the processing mode corresponding to the surgical scenespecified by the scene recognition result (surgical scene discriminationresult), and then, the sensor control unit 162 and the image combiningunit 152 perform processing corresponding to the processing modedetermined by the processing mode determination unit 151. That is, anappropriate processing mode corresponding to the surgical scene isdetermined by the correspondence table (for example, the correspondencetable in FIG. 4 or FIG. 6) and the processing corresponding to theprocessing mode is performed, making it possible to provide an optimalvideo image to the operator in accordance with the surgical scenes.

4. Modification

(Determination of Processing Mode by Planning)

While the above description is a case where the processing mode isdetermined in accordance with the external signal or the surgical scenespecified by the scene recognition, the processing mode determinationmethod may include, for example, a method of determining an appropriateprocessing mode in accordance with a surgery plan created beforehand.Specifically, for example, it is allowable to perform preliminaryplanning of the details of a surgery using a three-dimensional model toassociate each of surgical sites (affected sites) with a processing mode(for example, associating the high definition mode with a treatment fora certain affected site). Under this planning, in a case where futureimplementation of the treatment for the target surgical site (affectedsite) is detected from an output image (intraoperative image), it issufficient to shift to the processing mode associated with the surgicalsite (affected site).

5. Configuration of Computer

A series of processing (for example, image combining processing)described above can be executed in hardware or with software. In a casewhere the series of processing is executed with software, a programincluded in the software is installed in a computer. Herein, thecomputer includes, for example, a computer incorporated in a dedicatedhardware, and a general-purpose personal computer on which various typesof functions can be executed.

FIG. 12 is a block diagram illustrating an exemplary configuration ofhardware of a computer on which the series of processing described aboveis executed by a program.

In a computer 200, a central processing unit (CPU) 201, a read onlymemory (ROM) 202, a random access memory (RAM) 203 are interconnectedwith each other via a bus 204. The bus 204 is further connected with aninput/output interface 205. The input/output interface 205 is connectedwith an input unit 206, an output unit 207, a recording unit 208, acommunication unit 209, and a drive 210.

The input unit 206 includes a key board, a mouse, a microphone, and thelike. The output unit 207 includes a display, a speaker, and the like.The recording unit 208 includes hardware, a non-volatile memory, and thelike. The communication unit 209 includes a network interface and thelike. The drive 210 drives a removable medium 211 including a magneticdisk, an optical disk, a magneto-optical disk, a semiconductor memory,or the like.

On the computer 200 configured as above, the series of above-describedprocessing is executed by operation such that the CPU 201 loads, forexample, a program stored in the recording unit 208 onto the RAM 203 viathe input/output interface 205 and the bus 204 and executes the program.

The program executed by the computer 200 (CPU 201) can be recorded, forexample, in the removable medium 211 such as a package medium and beprovided. Alternatively, the program can be provided via a wired orwireless transmission medium including a local area network, anInternet, and digital satellite broadcasting.

On the computer 200, the program can be installed in the recording unit208 via the input/output interface 205, by attaching the removablemedium 211 to the drive 210. In addition, the program can be received atthe communication unit 209 via a wired or wireless transmission mediumand be installed in the recording unit 208. Alternatively, the programcan be installed in the ROM 202 or the recording unit 208 beforehand.

Note that the program executed by the computer 200 may be a programprocessed in a time series in an order described in the presentdescription, or can be a program processed in parallel or in a requiredtiming such as being called.

Note that in the present description, processing steps describing aprogram required for causing the computer 200 to execute various typesof processing are not necessarily processed in sequentially in an orderdescribed in the flowchart. The processing steps may include stepsexecuted in parallel or individually (for example, parallel processingor processing by objects).

In addition, the program can be processed by one computer or can behandled with distributed processing by a plurality of computers.Furthermore, the program can be transferred to a remote computer and beexecuted.

Furthermore, in the present description, the system represents a set ofmultiple constituents (devices, modules (parts), or the like). In otherwords, all the constituents may be in a same housing but they do nothave to be in the same housing. Accordingly, a plurality of apparatuses,housed in separate housings, connected via a network can be a system. Anapparatus in which a plurality of modules is housed in one housing canalso be a system.

Note that embodiments of the present technology are not limited to theabove-described embodiments but can be modified in a variety of wayswithin a scope of the present technology. For example, the presenttechnology can be configured as a form of cloud computing in which onefunction is shared in cooperation for processing among a plurality ofdevices via a network.

Moreover, each of steps described in the above flowcharts can beexecuted on one apparatus or shared by a plurality of apparatuses forprocessing. Furthermore, in a case where one step includes a pluralityof stages of processing, the plurality of stages of processing includedin the one step can be executed on one apparatus or can be shared by aplurality of apparatuses.

Note that the present technology can be configured as follows.

(1) An information processing apparatus including:

a processing mode determination unit that determines, in accordance withsurgical scenes, a processing mode for an in-vivo image captured by animaging apparatus including an imaging element arranged so as to enablepixel shift processing; and a processing unit that processes an imageoutput from the imaging apparatus, in accordance with the processingmode.

(2) The information processing apparatus according to (1),

in which the imaging apparatus includes a control unit that controls theimaging element in accordance with the processing mode, and

the processing mode determination unit supplies the processing mode tothe control unit.

(3) The information processing apparatus according to (1) or (2),

in which the processing mode includes a mode capable of providing, bythe pixel shift processing, a high definition image having higherdefinition than in the in-vivo image captured by the imaging element.

(4) The information processing apparatus according to (3),

in which the processing mode further includes at least one of a modecapable of providing an image with less noise than in the in-vivo imagecaptured by the imaging element and a mode capable of providing an imagein which motion appears smoother than in the in-vivo image captured bythe imaging element.

(5) The information processing apparatus according to any one of (1) to(3), in which the imaging apparatus includes at least three imagingelements.

(6) The information processing apparatus according to any one of (1) to(5),

in which the mode determination unit determines the processing mode inaccordance with an external signal input from an outside.

(7) The information processing apparatus according to (6), in which themode determination unit determines the processing mode in accordancewith operation performed by the operator to switch the processing mode.

(8) The information processing apparatus according to (6),

in which the mode determination unit determines the processing mode inaccordance with a signal indicating power on-off of a treatment tool,output from a treatment tool apparatus.

(9) The information processing apparatus according to (6),

in which the mode determination unit determines the processing mode inaccordance with a light source switching signal output from a lightsource apparatus.

(10) The information processing apparatus according to (1),

in which the mode determination unit determines the processing mode inaccordance with a surgery plan created beforehand.

(11) The information processing apparatus according to (10),

in which the mode determination unit determines the processing mode inaccordance with the processing mode for each of surgical sites plannedbeforehand in the surgery plan.

(12) The information processing apparatus according to any one of (1) to(5), further including a scene recognition unit that recognizes apredetermined scene on the basis of the image output from the imagingapparatus,

in which the mode determination unit determines the processing mode onthe basis of a recognition result obtained by the scene recognitionunit.

(13) The information processing apparatus according to (4),

in which the imaging apparatus includes a control unit that controls theimaging element in accordance with the processing mode,

the processing mode includes:

a first mode in which pixel shift processing is not performed and thein-vivo image captured by the imaging element is output at a constantframe rate;

a second mode in which pixel shift processing is performed and an imageobtained by the pixel shift processing and having higher definition thanin the in-vivo image captured by the imaging element is output at aframe rate same as a frame rate in the first mode;

a third mode in which pixel shift processing is not performed and animage obtained by a plurality of shots of imaging within a predeterminedtime is output at a frame rate higher than in the first mode; and

a fourth mode in which pixel shift processing is not performed andimages obtained by a plurality of shots of imaging within apredetermined time are added and then output at a frame rate same as theframe rate in the first mode, and

a surgical scene suitable for providing one of an image having higherdefinition than in the in-vivo image captured by the imaging element andan image with less noise than in the in-vivo image captured by theimaging element is associated with one of the second mode and the fourthmode, and a surgical scene suitable for providing an image in whichmotion appears smoother than in the in-vivo image captured by theimaging element is associated with the third mode.

(14) The information processing apparatus according to (4),

in which the imaging apparatus includes at least three imaging elements,

the processing mode includes:

a first mode in which each of RGB planes is imaged at a same timing andan image obtained by RGB plane images and having higher definition thanin the in-vivo image captured by the imaging element is output at aconstant frame rate; and

a second mode in which each of the RGB planes is imaged at differenttimings and an image obtained by combining RBG plane images having closeimaging times among the RBG plane images captured with shifted imagingtimings is output at a higher frame rate than in the first mode, and

a surgical scene suitable for providing an image having higherdefinition than in the in-vivo image captured by the imaging element isassociated with the first mode, and a surgical scene suitable forproviding an image in which motion appears smoother than in the in-vivoimage captured by the imaging element is associated with the secondmode.

(15) The information processing apparatus according to any one of (1) to(14),

in which the processing mode determination unit stores a tableassociating the surgical scene with the processing mode, and

determines the processing mode with reference to the table.

(16) The information processing apparatus according to (3),

in which the high definition image is an image having a horizontalresolution of 3840 or more.

(17) An information processing method of an information processingapparatus, the method including steps of:

determining, by the information processing apparatus in accordance withsurgical scenes, a processing mode for an in-vivo image captured by animaging apparatus including an imaging element arranged so as to enablepixel shift processing; and

processing an image output from the imaging apparatus, by theinformation processing apparatus in accordance with the processing mode.

(18) An endoscope system including an endoscope and an informationprocessing apparatus,

in which the endoscope includes:

an imaging element arranged so as to enable pixel shift processing; and

a control unit that controls the imaging element,

the information processing apparatus includes:

a processing mode determination unit that determines a processing modefor an in-vivo image captured by the endoscope, in accordance withsurgical scenes; and

a processing unit that processes an image output from the endoscope, inaccordance with the processing mode, and

the control unit controls the imaging element in accordance with theprocessing mode.

REFERENCE SIGNS LIST

-   10 Endoscopic surgery system-   101 CCU-   102 Light source apparatus-   103 Treatment tool apparatus-   104 Pneumoperitoneum apparatus-   105 Display apparatus-   106 Recorder-   107 Printer-   111 Endoscope-   112 Energy treatment tool-   121 Foot switch-   131 to 134 Trocar-   135 Forceps-   151 Processing mode determination unit-   152 Image combining unit-   153 Scene recognition unit-   161 Sensor-   161-1 Single-chip sensor-   161-3 Three-chip sensor-   162 Sensor control unit-   181-R, 181-G, 181-B Buffer-   182 Misalignment detection unit-   183 Misalignment correction unit-   200 Computer-   201 CPU

The invention claimed is:
 1. An information processing apparatuscomprising: processing circuitry configured to determine, in response toan external signal that includes information identifying one of aplurality of surgical scenes and includes a light source switchingsignal output from a light source apparatus, a processing mode for anin-vivo image captured by an imaging apparatus including an imagingelement configured to enable pixel shift processing; and process animage output from the imaging apparatus, in accordance with thedetermined processing mode, wherein the pixel shift processing isselectively enabled based on the processing mode, wherein the determinedprocessing mode is determined as a first mode when the light sourceswitching signal has a first value and is determined as a second modewhen the light source switching signal has a second value, and whereinthe first and second modes are different and the first and second valuesare different.
 2. The information processing apparatus according toclaim 1, wherein the imaging apparatus includes control circuitryconfigured to control the imaging element in accordance with theprocessing mode, and the processing circuitry is configured to supplythe processing mode to the control circuitry.
 3. The informationprocessing apparatus according to claim 1, wherein the processing modeincludes a mode capable of providing, by the pixel shift processing, ahigh definition image having higher definition than in the in-vivo imagecaptured by the imaging element.
 4. The information processing apparatusaccording to claim 3, wherein the processing mode further includes atleast one of a mode capable of providing an image with less noise thanin the in-vivo image captured by the imaging element or a mode capableof providing an image in which motion appears smoother than in thein-vivo image captured by the imaging element.
 5. The informationprocessing apparatus according to claim 3, wherein the imaging apparatusincludes at least three imaging elements.
 6. The information processingapparatus according to claim 1, wherein the processing circuitry isconfigured to determine the processing mode in accordance externalsignal input from an outside.
 7. The information processing apparatusaccording to claim 1, wherein the processing circuitry is configured todetermine the processing mode in accordance with an operation performedby an operator to switch the processing mode.
 8. The informationprocessing apparatus according to claim 1, wherein the processingcircuitry is configured to determine the processing mode in accordancewith a signal indicating power on-off of a treatment tool, output from atreatment tool apparatus.
 9. The information processing apparatusaccording to claim 1, wherein the processing circuitry is configured todetermine the processing mode in accordance with a surgery plan createdbeforehand.
 10. The information processing apparatus according to claim9, wherein the processing circuitry is configured to determine theprocessing mode in accordance with the processing mode for each ofsurgical sites planned beforehand in the surgery plan.
 11. Theinformation processing apparatus according to claim 1, wherein theprocessing circuitry is configured to recognize a predetermined scenebased on the image output from the imaging apparatus, and determine theprocessing mode based on the predetermined scene.
 12. The informationprocessing apparatus according to claim 4, wherein the imaging apparatusincludes control circuitry configured to control the imaging element inaccordance with the processing mode, the processing mode is one of aplurality of processing modes that includes: a first mode in which thepixel shift processing is not performed and the in-vivo image capturedby the imaging element is output at a constant frame rate; a second modein which the pixel shift processing is performed and an image obtainedby the pixel shift processing and having higher definition than in thein-vivo image captured by the imaging element is output at a same framerate as a frame rate in the first mode; a third mode in which the pixelshift processing is not performed and an image obtained by a pluralityof shots of imaging within a predetermined time is output at a framerate higher than in the first mode; and a fourth mode in which the pixelshift processing is not performed and images obtained by a plurality ofshots of imaging within a predetermined time are added and then outputat the same frame rate as the frame rate in the first mode, and asurgical scene suitable for providing one of an image having higherdefinition than in the in-vivo image captured by the imaging element andan image with less noise than in the in-vivo image captured by theimaging element is associated with one of the second mode and the fourthmode, and a surgical scene suitable for providing an image in whichmotion appears smoother than in the in-vivo image captured by theimaging element is associated with the third mode.
 13. The informationprocessing apparatus according to claim 4, wherein the imaging apparatusincludes at least three imaging elements, the processing mode is one ofa plurality of processing modes that includes: a first mode in whicheach of RGB planes is imaged at a same timing and an image obtained byRGB plane images and having higher definition than in the in-vivo imagecaptured by the imaging element is output at a constant frame rate; anda second mode in which each of the RGB planes is imaged at differenttimings and an image obtained by combining RBG plane images having closeimaging times among the RBG plane images captured with shifted imagingtimings is output at a higher frame rate than in the first mode, and asurgical scene suitable for providing an image having higher definitionthan in the in-vivo image captured by the imaging element is associatedwith the first mode, and a surgical scene suitable for providing animage in which motion appears smoother than in the in-vivo imagecaptured by the imaging element is associated with the second mode. 14.The information processing apparatus according to claim 1, wherein theprocessing circuitry is configured to store a table associating the oneof the surgical scenes with the processing mode, and determines theprocessing mode with reference to the table.
 15. The informationprocessing apparatus according to claim 3, wherein the high definitionimage is an image having a horizontal resolution of 3840 pixels or more.16. An information processing method of an information processingapparatus, the method comprising: determining, by the informationprocessing apparatus in response to an external signal that includesinformation identifying one of a plurality of surgical scenes andincludes a light source switching signal output from a light sourceapparatus, a processing mode for an in-vivo image captured by an imagingapparatus including an imaging element configured to enable pixel shiftprocessing; and processing an image output from the imaging apparatus,by the information processing apparatus in accordance with thedetermined processing mode, wherein the pixel shift processing isselectively enabled based on the processing mode, wherein the determinedprocessing mode is determined as a first mode when the light sourceswitching signal has a first value and is determined as a second modewhen the light source switching signal has a second value, and whereinthe first and second modes are different and the first and second valuesare different.
 17. An endoscope system including an endoscope and aninformation processing apparatus, wherein the endoscope includes: animaging element configured to enable pixel shift processing; and controlcircuitry configured to control the imaging element, the informationprocessing apparatus includes: processing circuitry configured todetermine a processing mode for an in-vivo image captured by theendoscope, in response to an external signal that includes informationidentifying one of a plurality of surgical scenes and includes a lightsource switching signal output from a light source apparatus; andprocess an image output from the endoscope, in accordance with thedetermined processing mode, the control circuitry is configured tocontrol the imaging element in accordance with the processing mode, andthe pixel shift processing is selectively enabled based on theprocessing mode, wherein the determined processing mode is determined asa first mode when the light source switching signal has a first valueand is determined as a second mode when the light source switchingsignal has a second value, and wherein the first and second modes aredifferent and the first and second values are different.